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INTRODUCTION 


Two earlier papers by R. B. MacDonald have described some of the 
research being pursued at LARS/Purdue directed toward the applications 
of remote sensing. The purpose of this presentation is to describe 
research in the development of new technology. Several projects to be 
discussed are of several years duration; current progress will be des- 
cribed in these cases. In addition, several new programs have begun 
and preliminary results will be presented. 


THE REGISTRATION OF MULTI SPECTRAL AND MULTITEMPORAL IMAGES 


The first study to be described has to do with the registration 
of multispectral images. This work began several years ago and has 
passed through graded steps of increasing technological difficulty. 

The ultimate objective is to develop a capability for accurately align- 
ing data in image form gathered from different parts of the spectrum 
and on different flight missions. 

This first step in this process several years ago was to achieve 
the capability to register image data from two different sensors 
mounted aboard the same platform. Figure 1 serves to illustrate the 
problem. Shown is a printout from a channel of data in the visible 
portion of the spectrum (center) and data gathered at the same time 
but with a different scanner operating in the thermal region. A con- 
ventional panchromatic air photo is shown on the left for comparison. 
Registration accuracy to better than plus or minus 1 resolution ele- 
ment is desired. 
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Since the two scanners may be running at slightly different 
speeds, and creating other distortions of a local nature, a scheme 
which assures correct registration on both a global and local basis 
is necessary. The general approach chosen was to use two-dimensional 
correlation as a basis for finding points of correspondence in the two 
images. This approach provides the maximum flexibility which is re- 
quired by a research situation as compared to an operational one. How- 
ever, it quickly became apparent that images from widely separated parts 
of the spectrum may not correlate well since the spectral separation 
leads to fundamental differences in the data. It was found desirable 
to preceed the correlation process by enhancing all the boundaries in 
both images and then correlating to the boundary-enhanced image. 

Thus, the first useful system contained three steps: a boundary 

enhancement step; a two-dimensional correlation to determine the points 
of coincidence in the two images; and finally, the overlaying of the 
two data sets to form a new data set of dimensionality equal to the sum 
of the previous two. 

The next step in the development of the system was to provide it 
with a measure of adaptability to the type of data being registered. 

The registration process utilizing two-dimensional correlation as a 
basis is clearly very scene-dependent. A different sized correlation 
window is required when registering data over an agricultural scene, 
for example, as compared to data from natural vegetation. Figure 2 
shows the organization of the system after incorporating this capability. 
A means for estimating the complexity of the image in terms of the degree 
of difficulty in obtaining a proper two-dimensional correlation measure 
has been added to the system. This results in a picture complexity index 
from which the correlation window size is determined. Developments to 
this point were reported in this meeting a year ago (see reference 1). 

Since that time, a number of additional capabilities have been 
added to the system. Chief among these has been the development of a 
capability for accomplishing rotation and scale change in the imagery. 

The current system diagram is shown in Figure 3 (see reference 2). The 
system now is organized to permit the overlay of two "slave" images onto 
a master. There is an opportunity to input scale and rotational correc- 
tions of a global nature manually into the system. After suitable buff- 
ering, the image boundary enhancement and complexity estimation step is 
next. Then follows the image correlation step in which the points of 
correspondence between the three images are determined. At this point 
the more precise local scale and rotational correction factors are de- 
termined and a capability for updating the global scale and rotation 
factors is provided. The final step is that of utilizing the scene cor- 
respondence points to achieve a single data tape on which have been 
properly registered the three previous images. 
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Two different methods of boundary enhancement techniques are 
available. Figure 4 shows examples of each. The simplest and com- 
putationally fastest means is a gradient technique utilizing the mag- 
nitude of the first difference of adjacent scene points to determine 
if a boundary exists. The result of such a computation is shown in 
the center of Figure 4. On the right is shown the result of using a 
clustering technique operating in a multivariant fashion to determine 
boundaries (see reference 3). 


The use of Temporal Information 

The new capability which the modifications have provided enabled 
initial studies in the use of multi-temporal multi-spectral data anal- 
ysis. Figure 5 shows printouts of single channels of the two data sets 
utilized. On the left is a printout of a channel of data from the June, 
1969 mission. Due to a cross wind, an average yaw distortion of 11.7 
degrees exists in the data. The August, 1969 data shown on the right 
contains very little yaw distortion. It is readily apparent from these 
two printouts that changes in the scene have taken place between June 
and August, and this provides a graphic example of the considerable 
scene dependence which exists in attempting to use two dimensional cor- 
relation directly for registration of images. Note in the June data 
that since the vegetation canopy does not fully cover the ground soil 
patterns are readily apparent in this display; by August this is no 
longer the case. 

In the multi- temporal analysis experiment, a control classification 
was run utilizing the June data only. A classifier was trained using 
classes com, soybeans, wheat and oats. The best four of the 12 channels 
of data were selected for this classification. Data from the August 
mission was registered onto the June data. The classification using the 
same classes, training, and spectral bands of this August data was then 
run for comparison. Next, the best four of the 8 channels (4 from June, 
the same 4 from August) were selected and the same classification was 
again carried out. The results of this experiment are shown in Figure 6. 
Note that the capability to discriminate between corn and soybeans does 
improve over that available from either of the data sets alone. It 
should also be noted that by the August mission all wheat and oats in the 
flightline had been harvested; these classes therefore did not exist by 
the time of that mission. 

This classification test, while preliminary in nature, does tend to 
verify the expected increase in accuracy one should obtain by having 
multi-temporal data available. It seems reasonable to speculate that 
even greater accuracy should be possible if one permits oneself to select 
the best four of the total number of channels available (24 in this case) 
rather than constraining to the spectral bands which were favored for the 
June data set. The above procedure was used, however, in order to simu- 
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late the situation which might exist for a satellite sensor system in 
which the spectral bands to be used throughout must be selected a priori. 


On the Value of Image Data Registration 

There are at least four major reasons why image data registration 
is important. These are indicated in Figure 7. The desirability of 
registration in the case of multiple sensors has already been discussed. 
This was discussed within the context of visible and thermal data; how- 
ever, the problem is similar in the case of data from the microwave 
portion of the spectrum. In general, each new part of the spectrum 
available in registered form should provide additional information about 
the contents of the scene. 

The above example tended to illustrate the importance of the use of 
temporal information. However, not nearly enough experimentation has 
been possible to date in order to develop the full importance of this 
area. 

A third reason for the importance of image data registration is 
that it permits the automatic correction of various types of distortion 
in image data. If a good quality image (from a photogrammetric stand- 
point) exists of an area, this image could be made to serve as a master 
upon which data from other sensors e.g. scanner, radar, television, etc. 
could be registered; in this fashion, rectification and geometric cor- 
rection of all types could take place as a normal part of achieving 
multiband multi-temporal information. 

Finally, the through-put rate for this type of information process- 
ing is limited as much as anywhere at points at which ground data and 
ancillary information must be merged into the multispectral image data 
stream itself. Data registration can, in many cases, tend to alleviate 
this difficulty. For example, in the case of an agricultural problem, 
one need only establish field boundaries etc. once during a growing 
season in order to have ground truth information referenced to the appro- 
priate fields in this image data. Since the registration process estab- 
lishes scene point coincidences In the images it established this same 
coincidence and referencing for all ground truth associated with given 
scene points. Thus, training sample coordinates need be selected only 
once during a season. 


DIGITAL IMAGE DISPLAY SYSTEM 


A second project underway for some time is the specification, design, 
procurement, and implementation of a digital image display system. A 
pacing item in the development of a capability to rapidly collect, ana- 
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lyze and disseminate information by remote sensing means is the capa- 
bility for the human operator to rapidly interact with the image data 
stream and collate with it literal or other information from other 
sources. The purpose of the digital image display is to enable re- 
search into techniques to alleviate this limitation. 

The need for such a system and the general outline of its specifi- 
cations was first identified in March, 1966. After a long series of 
delays, the project was finally funded and a contract awarded for its 
construction by the IBM Federal Systems Division in August, 1969. The 
system has recently arrived on location at Purdue and is currently 
undergoing acceptance tests. It consists of three major elements: a 

control unit containing logic circuitry, interfacing circuitry, and a 
disk image buffer; the display console including a light pen and function 
keyboard for controlling software packages, and a photocopy unit used 
to generate hard copy versions of images being displayed. 

Figure 8 shows a photograph of the photocopy unit (right) and dis- 
play console (left). The display system provides an image of 768 ele- 
ments and 577 lines with a 16-step grayscale and a 30-frame-per-second 
(with interlace) refresh rate. Figure 13 shows the output of image data 
produced by the photocopy unit. Upon final acceptance of the system 
from the manufacturer, work will begin on techniques for allowing the 
human operator to use very large quantities of remote sensing data. 


DATA SYSTEMS PARAMETER STUDY 

Figure 9 is a block diagram of a satellite-based remote sensing 
system. Shown are the major elements of such a system, such as the 
sensor, onboard processing and telementry , ground processing and data 
reduction, and information consumption. In considering the design of 
such a system, it is immediately apparent that there are a great number 
of parameters to be chosen. Parameters such as the spectral and spatial 
resolution of the sensor, the detector signal-to-noise ratio, the type 
of telemetry, data compression scheme, etc. must all be selected. 

Heretofore, research on individual elements of the system had not 
proceeded far enough to permit the consideration of overall performance 
relative to these parameters. It now appears, however, that this point 
has been reached. As a result, a data system parameter study has been 
defined. The approach to be used is to test the overall system sensi- 
tivity to various key parameters. Generally, the index of performance 
to be used is the pattern recognition classification performance on test 
data sets. Preliminary results in several of these parameter studies 
will now be described. 
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Classification Sensitivity to Additive Noise 

A parameter of considerable importance to the design of new 
sensor systems is the signal-to-noise ratio of the detector. For a 
sensor material of given sensitivity the signal-to-noise ratio of the 
data produced can generally be improved by degrading either the spec- 
tral or spatial resolution or both. However, data analysts are gen- 
erally not anxious to see this done and a trade-off must be made. 
Quantitative information about desired signal-to-noise ratios for 
specific analysis tests has previously been scarce. 

In order to increase the understanding of the necessary signal-to- 
noise ratio for various classification tests it was decided to carry 
out a single test classification on a data set to which varying amounts 
of noise have been added. Figure 10 shows a small portion of a scene 
from which the data was selected. Shown are a panchromatic air photo 
of the area and below it a printout of the original undegraded signal. 
Printouts of the same data with two different levels of noise added 
are shown on the right. Digital data of eight-bit precision was used. 
Thus, a gray scale of 256 possible steps is available. The magnitude 
of the noise (which was Gaussian and uncorrelated between channels) is 
measured in terms of the number of bins out of 256 per standard deviation. 

The noise was generated using a software-implemented random number 
generator within the computer. It was decided to choose Gaussian noise, 
uncorrelated from spectral band to spectral band, as this most nearly 
approximates the type of noise generated within the sensor detector. 

Figure 11 is a classification of a segment of the data for the no-noise 
case and the two noise levels shown in Figure 10. The degradation of 
accuracy is visibly apparent. 

The graph of Figure 12 summarizes the overall results for the study. 
This is a plot of the percent correct recognition for the classification 
task as a function of the magnitude of noise added to the data. The 
shapes of the curves which are in the form of the complement of the 
Gaussian error function could easily be predicted. It is also not sur- 
prising that the results for the training samples stayed consistently 
above the results for the overall test samples. 

An interesting result can be seen from the two individual classes 
which are plotted. The data used for this test was of an agricultural 
scene in Tippecanoe County in June, early in the growing season. At 
this point in the season wheat is ready for harvest and, being golden 
brown at this stage, is a relatively easy class to separate from the 
rest of the scene. Thus, in the graph of Figure 12 the accuracy is 
high. More importantly at this point though, it degrades relatively 
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slowly. On the other hand, soybeans provides a much more challenging 
classification since the percent of ground cover at this stage of the 
season is very low. While the accuracy in the no-noise case is quite 
high, one sees that the additive noise quickly degrades this accuracy 
to a low value. These results tend to bear out the statement that as 
the signal-to-noise ratio degrades, the more difficult classifications 
will be affected most. That is, for simple classifications the signal- 
to-noise ratio is not too important; however, it becomes so for more 
difficult ones. 

The classification used was a nine class classification. One would 
assume that in an extrapolation to the right, these curves would become 
asymptotic to chance performance which is 11 percent in this case. One 
can also consider an extrapolation of these curves to the left. This 
extrapolation should provide curves which become asymptotic to 100 per- 
cent accuracy. Since the original data did not have an infinite signal- 
to-noise ratio, it may be supposed that these curves provide an indi- 
cation of the true original signal-to-noise ratio. 


Data Compression based on Spatial Redundancy 

The General Electric Company was awarded a contract some time ago 
by NASA to study the possibility of using data compression techniques 
on a sensor system of the type to be flown on the Earth Resources Tech- 
nology Satellite. The techniques to be studied apparently are to be 
those based primarily on the fact that a great deal of redundancy exists 
in an image due to spatial correlation in the data. As a result of the 
application of efficient data compression techniques, it is possible to 
reconstruct the image transmitted to good but not perfect precision. 

It is, therefore, desirable to learn the degree to which the data com- 
pression technique will affect the ability of the data analyist to 
achieve good results . 

It is desirable to determine these effects on both photointerpre- 
tative based analysis systems and pattern recognition (machine oriented) 
schemes. A co-operative program between LARS and GE was agreed upon in 
order to accomplish a portion of this evaluation. Test data was selected 
jointly by LARS and GE from among flightlines of data which had been 
flown by the Michigan scanner system and digitized at LARS. The data 
was provided to GE for the purpose of compressing and re-expanding it 
using compression algorithms of General Electric Company design. The 
four channels of the Michigan scanner system most nearly coinciding with 
the four channels of the ERTS multispectral scanner were to be used. 

After compression and re-expansion the data was returned to LARS and a 
test classification run upon it. 
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The joint study has now progressed to the point that the first 
compression algorithm has been tested. Figure 13 shows images gener- 
ated from a small portion of the original data and the same data having 
been processed by the compression algorithm. The algorithm is a pre- 
liminary one for base line tests and is simply a procedure whereby only 
every fourth sample of each channel is transmitted. Intermediate points 
not transmitted are assumed on the ground to have been the same gray 
scale intensity as the last previously transmitted point in that spec- 
tral band. 

Figure 14 shows the results of the test classification on the two 
different data sets. Also shown is the results of the classification 
of the same data but using the set of spectral bands which are pre- 
ferred for this classification. Notice that the degradation provided 
by the compression as compared with the uncompressed data is not major. 
Full analysis of results must await the conclusion of the study includ- 
ing the use of other compression schemes. These will be reported in due 
course. However, some further comments about these results will be made 
after having reported the results of another data compression study in 
progress. 


DATA COMPRESSION TECHNIQUES BASED ON SPECTRAL REDUNDANCY 


A second data compression study has been underway for some time 
based on the use of the redundancy between spectral bands. This study 
is being conducted entirely by LARS. The point of departure of this 
study is somewhat different than the one previously described in that a 
larger number of spectral bands are assumed. An overview of the view 
point is contained in Figure 15. We see on the left a multispectral 
scanner sensor which produces data in M different channels. The tech- 
nique here would be to design a data compression system which converts 
the M channels into N features, N < M. The original data could be re- 
covered to within some prescribed accuracy by processing the data at 
this point through a reconstruction algorithm after which data analysis 
could take place. 

However, imagery could be generated in each of the N features in 
exactly the same way that it could be generated in each of the M chan- 
nels. In any case, the measure of the degree of compression in this 
case is the ratio of M to N. 

The compression algorithm tested is a type based upon a Karhunen- 
Loeve expansion. It amounts, in fact, to a linear transformation in M- 
dimensional space and, more specifically, a linear transformation which 
is a rotation to principal co-ordinates. The same data and classifi- 
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cation was used to test this scheme as used in the LARS/GE Data Compres- 
sion Study and the Additive Noise Study. The results of this classifi- 
cation are shown in Figure 16. The first line gives the results of the 
control classification, that is, classification directly on the original 
data. 


When the 12 original channels were transformed and then half deleted 
so that only six features remained in the N-dimensional feature space, 
the accuracy of the classification did not change significantly. The 
test was repeated for a 12- to 3-dimensional transformation and again 
the accuracy was essentially unaffected. Not until all but two bands 
in feature space were deleted did the accuracy begin to seriously deter- 
iorate. The percent of mean square difference between the reconstructed 
and the original image is shown on the right. These results have been 
judged to be very encouraging and the scheme is now being extended to 
include both spectral and spatial redundancy. 


SOME COMMENTS ON SPECTRAL BAND SELECTION 


Some of the results above raise some points that have to do with 
the relative importance of a band selection capability. Notice from 
Figure 14 that the overall classification accuracy for the preferred 
channels (i.e. the best of the 12 available) was approximately 90%. 

This is more than 10% higher than the performance figure for the same 
data but using the ERTS channels. Thus, one may say from the data 
analyst’s viewpoint that being required to give up the capability to 
tailor the band selection to the particuliar classification being carried 
out is a more serious effect than accepting the degrading effects in the 
data due to a data compression algorithm. This can be seen and under- 
stood more clearly as follows: 

Assume this case to be typical. Here, 12-channel data was available; 
the band limits of the particular bands are shown in Figure 17. The pro- 
blem of band selection comes to finding the four best channels^ of the 12 
to use for the particular classification. The feature selection algorithm 
implemented in the LARSYSAA programming system computes the relative sep- 
arability of each class pair for each possible four-tuple of spectral bands. 
The classes used in the test classification are shown on Figure 18. Beside 
each is indicated a single syinbol used to designate that class in two 
following figures. Figure 19 shows the results of applying this algorithm 
to this classification task. The numbers on the right of this figure are 
the numbers indicating the relative separability of the class pairs. 

Class pairs are indicated by two symbols (SC for soybeans and corn, SW 
for soybeans and wheat) at the head of each of these columns. The four- 


^A preliminary determination as to the number of channels to be used must, 
of course, be made. Four have been assumed. 
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tuple of features (spectral bands) are then rank-ordered based upon 
the average of these interclass separability measures. Thus, it can 
be seen that bands 1, 9, 11 and 12 were selected as the best feature 
set and bands number 6, 9, 11 and 12 which are the ones most nearly 
matching the ERTS channels are second. 1 

However, by using additional options available with the feature 
selection algorithm it is possible to further tailor the band selection 
to the classification task. Note that in Figure 19 some of the inter- 
class pair separabilities are very large, indicating very obvious sep- 
arability; on the other hand, other interclass separability measures 
are quite small. It would be desirable to select feature sets so as 
to increase the separability of the more difficult interclass pairs 
at the expense of the classes with the more obvious separability. In 
order to do this the feature selection algorithm has been provided with 
an option permitting the imposing of a maximum interclass separability 
measure which will be considered for the purpose of rank-ordering the 
four-tuples. Figure 20 shows the results of using a maximum of 200. 

Note that the preferred feature set now becomes 1, 6, 10 and 12 and that 
the ERTS simulated channels 6, 9, 11 and 12 become 55th in ranking. 

The validity of this re-ranking is borne out in the difference in 
overall accuracy ultimately obtained in the two classifications. Approx- 
imately 90% for the preferred channels 1, 6, 10 and 12 as compared to 
approximately 80% for the ERTS channels 6, 9, 11 and 12. 2 

Thus, by being forced to a sub-optimum choice of spectral bands an 
overall 10% accuracy loss occured and the increased loss in accuracy 
due to data compressions was only an additional 2 or 3 percent. 

These results tend to suggest aht the capability to make a proper 
selection of sepctral bands from a large set may indeed be far more 
important than the effects of a perhaps lower signal-to-noise ratio, a 
data compression scheme, and conceivably other system parameters. As a 
matter of fact the scheme indicated in Figure 15 whereby in the space- 
craft a scanner with many channels is operating but a compression algor- 
ithm reduces the dimensionality for transmission to earth may provide a 
more useful approach. Further investigations into this type of scheme 
are under way. 


In the case of an actual satellite however, channel 1 would probably not 
be as useful as it appears to be here, since it is well into the blue por- 
tion of the spectrum and from space there would be considerable blue 
scattering. These results tend to bear out the choice that the 4 ERTS 
spectral bands are generally a good set. 

2 

From these results one might tend to conclude that the ERTS channels 
which are the best set of bands In general may be considerably sub- 
optimum in specific cases . 
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ADAPTIVE MULTISPECTRAL PICTURE 
REGISTRATION SYSTEM 


Figure 2 
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DIGITAL IMAGERY REGISTRATION SYSTEM 


Figure 3 
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Border Enhancement Techniques 


Air 

Photo 



Gradient 

Borders 

Figure 1+ 
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Borders 
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June 1969 
11.7® Yaw Distortion 



August 1969 
0® Yaw Distortion 


Flightline PF24 
Attitude 5000 ft. 

Band ,66-72 microns 


Example of Rotational Distortion in 
Multispectral Airborne Scanner Imagery 


Figure 5 
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MULTI -TEMPORAL ANALYSIS 
Scanner Data from June & August, 1969 
PF24 Run 69004801 
Test Field Classification Accuracy 




June 

(Ch 3,6,9,12) 

August 

(3,6,9,12) 

June /Aug. 

(6,9,12 June; 12 Aug.) 


No. of 
Samples 




Corn 

3281 

80.2% 

53.6% 

86.4% 

Soybeans 

1738 

83.3% 

89.6% 

91.8% 

Wheat 

396 

71.5% 


75.5% 

Oats 

693 

78.1% 


67.5% 

OVERALL 


80.3% 

574% 

85.8% 

Channel 

3 

Band 

.52 - 55/j 


'Best 4 of the 8 Available by the 
Divergence Analyser 


6 .62-.66/J 

9 .00-1.0/1 

12 2 . 0-2 


Figure 6 
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Image Data Registration 

• Non-single Aperature Sensors 

• Temporal Information 

• Distortion Correction 

• Ground -Data Merge 


Figure 7 • 
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Figure 8 
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Classification Results, Wheat Only 




Classification Performance vs Noise 



— Training Samples 
Test Samples 

Wheat (Test) 

Soybeans (Test) 
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GE/LARS Data Compression Study 



Original Data Compressed 


Figure 13 
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GE/LARS Data Compression Study 

June 1966 Cl Test Results 
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Wheat 

2641 



I 


Soy Deans Corn 
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Total Samples = (9246 
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MSS Data Compression 
Based on Spectral Correlation 



M N M 

Channels Features Channels 


Figure 15 
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MSS Data Compression 
Based on Spectral Correlation 

Compression Overall Mean Square 


Ratio M/N 

Performance 
in % 

Error in % 

12/12 

90.1 

0 

12/6 

88.6 

l.l 

12/3 

89.5 

2.2 

12/2 

60.7 

6.6 

Spectral Bands for 
and .80- 

Classification = .40- 
I.O/jm 

44, .52 - 55, .66-72 


Figure l6 


492 



1+0-28 


Spectral Band Selection 


Spectral Bands 
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9 
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10 
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II 
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6 

.52-55 

12 

.80-1.00 -ERTS 
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Figure 17 
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Spectral Band Selection 


Classes 
S - Soybeans 
0 - Oats 
C - Corn 
M- Wheat 2 


R - Red Clover 
A - Alfalfa 
W- Wheat I 
Y- Rye 
X - Bare Soil 


Figure 18 
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Class Separability Measure 
(No Maximum) 


Rank 

Spectral Bands 

Average Separability 

Individual 

Class Separability 




SC 

SW 

SA 

WR 

MY . 

1 

1, 9, 11, 12 

444 

25 

190 

188 

620 

58 

- 2 

6, 9, 11, 12 

428 

26 

177 

299 

630 

208 

3 

2, 9, 11, 12 

423 

24 

151 

182 

619 

58 

4 

5, 9, 11 , 12 

420 
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8, 9, 11 , 12 
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Figure 19 
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Class Senarability Measure 
(Maximum » 200) 


Rank 

Spectral Bands 

Averane Senarability 

InHi vidua 1 

Class 

Seoarabl 1 Ity 




sc 

sw 

SA 

UP 

WY 

1 

1 , 6, 10, 12 

155 

34 

200 

106 

200 

180 

o 

c 

1, 6, 10, 11 

154 

34 

200 

102 

200 

183 

3 

1,6,0, 12 

153 

26 

200 

193 

200 

200 

4 

1,6,0, 11 

153 

27 

200 

190 

200 

200 

ERTS- 55 

5, 9, ’ll , 12 

145 

26 

177 

200 

200 

200 


Figure 20 
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